{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets\n",
    "from torchvision.transforms import ToTensor\n",
    "\n",
    "training_data = datasets.FashionMNIST(\n",
    "    root=\"data\",\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")\n",
    "\n",
    "test_data = datasets.FashionMNIST(\n",
    "    root=\"data\",\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=ToTensor()\n",
    ")\n",
    "\n",
    "train_dataloader = DataLoader(training_data, batch_size=64)\n",
    "test_dataloader = DataLoader(test_data, batch_size=64)\n",
    "\n",
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.linear_relu_stack = nn.Sequential(\n",
    "            nn.Linear(28*28, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 512),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(512, 10),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.flatten(x)\n",
    "        logits = self.linear_relu_stack(x)\n",
    "        return logits\n",
    "\n",
    "model = NeuralNetwork()"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 超参数\n",
    "超参数是你用来控制模型优化过程的、可以调整的参数。不同的超参数取值能够影响模型训练和收敛的速度(更多关于调整超参数的内容)\n",
    "\n",
    "我们定义以下用于训练的超参数:\n",
    "\n",
    "Number of Epochs - 迭代数据集的次数\n",
    "Batch Size - 在参数更新之前通过网络传播的数据样本数量。\n",
    "Learning Rate - 学习率， 每 Batch/Epoch 次更新模型参数的幅度。较小的值会产生较慢的学习速度，较大的值可能会在训练过程中产生无法预料的行为。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "learning_rate = 1e-3\n",
    "batch_size = 64\n",
    "epochs = 5"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 优化循环\n",
    "设置完超参数后，接下来我们在一个优化循环中训练并优化我们的模型。优化循环的每次迭代叫做一个 Epoch(时期、纪元)。\n",
    "\n",
    "每个 Epoch 由两个主要部分构成:\n",
    "\n",
    "训练循环 在训练数据集上遍历，尝试收敛到最优的参数。\n",
    "验证/测试循环 在测试数据集上遍历，以检查模型效果是否在提升。\n",
    "\n",
    "\n",
    "## 损失函数\n",
    "拿到一些训练数据的时候，我们的模型不太可能给出正确答案。损失函数能衡量获得的结果相对于目标值的偏离程度，我们希望在训练中能够最小化这个损失函数。我们对给定的数据样本做出预测然后和真实标签数据对比来计算损失。\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "loss_fn = nn.CrossEntropyLoss()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 优化器\n",
    "优化是在每一个训练步骤中调整模型参数来减小模型误差的过程。优化算法定义了这个过程应该如何进行(在这个例子中，我们使用 Stochastic Gradient Descent-即SGD，随机梯度下降)。所有优化的逻辑都被封装在 optimizer 这个对象中。这里，我们使用 SGD 优化器。除此之外，在 PyTorch 中还有很多其他可用的优化器，比如 ADAM 和 RMSProp 在不同类型的模型和数据上表现得更好。\n",
    "\n",
    "我们通过注册需要训练的模型参数、然后传递学习率这个超参数来初始化优化器。"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [],
   "source": [
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "在训练循环内部, 优化在三个步骤上发生：\n",
    "\n",
    "- 调用 optimizer.zero_grad() 来重置模型参数的梯度。梯度会默认累加，为了防止重复计算(梯度)，我们在每次迭代中显式的清空(梯度累加值)。\n",
    "- 调用 loss.backward() 来反向传播预测误差。PyTorch 对每个参数分别存储损失梯度。\n",
    "- 我们获取到梯度后，调用 optimizer.step() 来根据反向传播中收集的梯度来调整参数。\n",
    "\n",
    "## 完整实现\n",
    "我们定义 train_loop 为优化循环的代码，test_loop 为根据测试数据来评估模型表现的代码"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def train_loop(dataloader, model, loss_fn, optimizer):\n",
    "    size = len(dataloader.dataset)\n",
    "    # Set the model to training mode - important for batch normalization and dropout layers\n",
    "    # Unnecessary in this situation but added for best practices\n",
    "    model.train()\n",
    "    for batch, (X, y) in enumerate(dataloader):\n",
    "        # Compute prediction and loss\n",
    "        pred = model(X)\n",
    "        loss = loss_fn(pred, y)\n",
    "\n",
    "        # Backpropagation\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        if batch % 100 == 0:\n",
    "            loss, current = loss.item(), (batch + 1) * len(X)\n",
    "            print(f\"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]\")\n",
    "\n",
    "\n",
    "def test_loop(dataloader, model, loss_fn):\n",
    "    # Set the model to evaluation mode - important for batch normalization and dropout layers\n",
    "    # Unnecessary in this situation but added for best practices\n",
    "    model.eval()\n",
    "    size = len(dataloader.dataset)\n",
    "    num_batches = len(dataloader)\n",
    "    test_loss, correct = 0, 0\n",
    "\n",
    "    # Evaluating the model with torch.no_grad() ensures that no gradients are computed during test mode\n",
    "    # also serves to reduce unnecessary gradient computations and memory usage for tensors with requires_grad=True\n",
    "    with torch.no_grad():\n",
    "        for X, y in dataloader:\n",
    "            pred = model(X)\n",
    "            test_loss += loss_fn(pred, y).item()\n",
    "            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n",
    "\n",
    "    test_loss /= num_batches\n",
    "    correct /= size\n",
    "    print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 2.301392  [   64/60000]\n",
      "loss: 2.293064  [ 6464/60000]\n",
      "loss: 2.272499  [12864/60000]\n",
      "loss: 2.263719  [19264/60000]\n",
      "loss: 2.258502  [25664/60000]\n",
      "loss: 2.219510  [32064/60000]\n",
      "loss: 2.225234  [38464/60000]\n",
      "loss: 2.190888  [44864/60000]\n",
      "loss: 2.188061  [51264/60000]\n",
      "loss: 2.146174  [57664/60000]\n"
     ]
    }
   ],
   "source": [
    "train_loop(train_dataloader,model, loss_fn, optimizer)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Error: \n",
      " Accuracy: 41.2%, Avg loss: 2.149597 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_loop(test_dataloader, model, loss_fn)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}